Skip to main content

View Post [edit]

Poster: aronsson Date: Oct 6, 2009 7:01am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

Thanks! These two items have now been checked in as
http://www.archive.org/details/wikipedia_visitor_stats_200902
and
http://www.archive.org/details/wikipedia_visitor_stats_200903

I'm still puzzled by the low throughput on FTP uploads. I never see more than 319 kbyte/s per session, but I can double that by running two parallel sessions. Is somebody looking at that?

Reply [edit]

Poster: Nemo_bis Date: Nov 15, 2010 12:22pm
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

> I'm still puzzled by the low throughput on FTP uploads. I
> never see more than 319 kbyte/s per session, but I can
> double that by running two parallel sessions. Is somebody
> looking at that?

I tried and used Flash but it's even worse: http://www.archive.org/post/333852/maximum-upload-speed-flash
Now I'm uploading with FTP: 9 parallel uploads (which seem to be the maximum, and yet enough to be often flooded with ECONNRESET error) are now at about 1,5 MiB/s on average, with peaks of 2 MiB/s and much more rarely 3 MiB/s, which may be the actual limit on my end (although it should be higher).

This post was modified by Nemo_bis on 2010-11-15 20:22:58

Reply [edit]

Poster: Nemo_bis Date: Nov 17, 2010 6:43am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

Well, now FileZilla has gone crazy for some minutes because after a "time expired for connection" error which aborted the uploads it couldn't recconnect because of ECONNRESET errors and a "421 Too many connections (10) from this IP" error; I've raised the time between failed login retries from 10 to 60 s and it has recovered after three minutes. Now trying with 8 parallel uploads.

Reply [edit]

Poster: tracey pooh Date: Oct 6, 2009 10:02am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

there can be massive contention for the disks with heavy fragmented reading especially at times. so the speed can be be very bad at times.

we have mostly given up on FTP technology here at archive for exactly the reasons you point out -- why is it going so slow when i can add a second uploader and get 2x the throughput? (that *is* interesting 8-) why are the disks not maxxed out fully by the ftp application? why does resumption not work? all these questions and more were unanswerable by myself and our staff so we have made our single remaining FTP "WIGIWYG", unfortunately ("what you get is what you get").

nonideal to be sure -- but our pure-ftpd server program has had so many problems that we've stopped trying to fix and/or patch around it.

another option is S3 (HTTP PUT) that we support...
-tracey

Reply [edit]

Poster: kzns.pl Date: Oct 26, 2009 8:56am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

Hi.
Is there a way to automatic upload of a file to add to an item I've already created? (ie use external program to upload)

We have a community radio (internet based) and are give a .wav file recorded by journalists, and local artists.

We have an in-house specialist computer program which with one-click takes the file, converts it to .mp3, adds ID3 tags, adds it to our database, converts it to text, and creates a webpage for it and (up to now) uploads the mp3 to a server (so the non-technical staff do not have to waste their time doing what a computer can do). Also it means technical staff can concentrate on more taxing issues than upload a file.

We would like to use the Internet Archive for this as it is the best or proper place for it for archiving purposes, but... what technical methods exist, however complex, for automating the process of getting that mp3 file from an inhouse computer to a published and accessible version on Internet Archive?

There were some methods floating around (using php) to upload uears ago when the FTP was all working well.


kzns.pl

Reply [edit]

Poster: tracey pooh Date: Nov 2, 2009 6:33pm
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

yes there is -- i can try to put you in touch w/ an engineer here who can point you to S3. you'd want something like curl (ideally) or wget on a unix-like cmd-line if that's possible. then it's relatively straightforward from there.

we can proceed if that sounds good/plausible and you reply!
-tracey

Reply [edit]

Poster: tracey pooh Date: Nov 2, 2009 6:30pm
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

yes there is -- i can try to put you in touch w/ an engineer here who can point you to S3. you'd want something like curl (ideally) or wget on a unix-like cmd-line if that's possible. then it's relatively straightforward from there.

we can proceed if that sounds good/plausible and you reply!
-tracey

Reply [edit]

Poster: aronsson Date: Oct 6, 2009 10:18am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

My friends mentioned the TCP window scaling option. With my ping roundtrip time of 194 ms (from 130.236.254.105 in Sweden) TCP without window scaling might max out at 320 kbyte/s.

It is on at my end. I'm running a standard Ubuntu Linux 9.04 and "cat /proc/sys/net/ipv4/tcp_window_scaling" answers "1" (meaning true/on).

Reply [edit]

Poster: Shiva Ho Date: Oct 13, 2009 10:16am
Forum: forums Subject: FLASH SUCKS

Your rational is flawed!!! FTP WORKS & Flash doesn't!!!!

Reply [edit]

Poster: tracey pooh Date: Oct 13, 2009 11:36am
Forum: forums Subject: Re: FLASH SUCKS

then don't use flash! 8-)

there is a "prefer to not use flash?" link
on the uploader (new item) page and
one on the edit item files (for an existing item) page.

--tracey

Reply [edit]

Poster: Shiva Ho Date: Oct 13, 2009 10:14am
Forum: forums Subject: Re: Stale .pureftpd-upload* file blocks checkin

The "NEW" system has even more flaws & times out repeatedly making it unusable...
Lets FACE it FLASH SUCKS ALWAYS HAS & ALWAYS WILL...
If you can't do it with straight html & java then stop wasting our time & Bandwidth